Skip to content

Conversation

ahgraber
Copy link
Contributor

@ahgraber ahgraber commented Jun 9, 2025

  • refactor: move is_event_loop_running and as_completed to async_utils
    This helps separate async features from executor and engine modules, and will prevent circular imports in these modules.
  • feat: add ProgressBarManager for enhanced progress tracking
    Introduces a new ProgressBarManager class to manage progress bars for both batch and non-batch execution. This includes methods for creating single and nested progress bars, as well as updating batch progress bars, improving user experience during long-running tasks.
  • feat: refactor async utilities for improved functionality
    • Refactored run function from repeated async, executor, and engine logic.
    • Changed as_completed from async to a regular function for better compatibility.
    • Added process_futures to handle futures with optional progress tracking.
  • feat: enhance async task management with ProgressBarManager
    • Refactored Executor to utilize new async_utils and ProgressBarManager for improved composition
    • Refactored run_async_tasks to utilize ProgressBarManager for improved progress tracking.
    • Modified tests to validate the new behavior functions
  • feat: enhance Executor with job indexing and exception handling
    • Added _jobs_processed attribute to maintain consistent job indexing across multiple runs.
    • Implemented clear_jobs method to reset job indices.
    • Updated tests to verify exception handling and job indexing after clearing jobs.
    • Adjusted Jupyter notebook tests for consistency in results retrieval.
  • feat: update engine's apply_transformations to recursively process nested Transformation or Parallel procedures
    • The apply_transforms() function has been updated to handle different types of transformations recursively:
      If transforms is a list, it recursively applies each transform in the list.
      If transforms is a Parallel instance, it recursively applies the transformations contained within the Parallel object.
      If transforms is a BaseGraphTransformation, it generates an execution plan (a list of coroutines), gets a description, and then runs the coroutines asynchronously using run_async_tasks().
      If transforms is none of the above, it raises a ValueError indicating an invalid type.
    • Move apply_nest_asyncio to async_utils for better organization
    • Updated the Parallel class to support transformations with improved type hints.
    • Added unit tests
  • chore: use Sequence over List for type checking improvements; add debug logging

@dosubot dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Jun 9, 2025
@ahgraber
Copy link
Contributor Author

ahgraber commented Jun 9, 2025

I know this one is a big review - everything's interrelated enough that I had a hard time splitting it up, but I could try to refactor into smaller pulls if necessary.

@ahgraber ahgraber changed the title Feature/async executor feature! improve async / executor functionality Jun 9, 2025
@ahgraber ahgraber changed the title feature! improve async / executor functionality feature: improve async / executor functionality Jun 9, 2025
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Jul 1, 2025
@ahgraber ahgraber force-pushed the feature/async-executor branch from ecfdfcd to d5bdcc3 Compare July 18, 2025 16:52
@anistark
Copy link
Contributor

anistark commented Sep 8, 2025

@ahgraber Thanks for the PR. Could you please try rebase and check for the conflicts?
Would be easier to review.

@ahgraber ahgraber force-pushed the feature/async-executor branch from c68f37c to 4b7774f Compare September 8, 2025 13:42
ahgraber and others added 16 commits September 8, 2025 10:02
This helps separate async features from executor and engine modules, and will prevent circular imports in these modules.
Introduces a new ProgressBarManager class to manage progress bars for both batch and non-batch execution. This includes methods for creating single and nested progress bars, as well as updating batch progress bars, improving user experience during long-running tasks.
- Refactored `run` function from repeated async, executor, and engine logic.
- Changed `as_completed` from async to a regular function for better compatibility.
- Added `process_futures` to handle futures with optional progress tracking.
- Refactored Executor to utilize new async_utils and ProgressBarManager for improved composition
- Refactored run_async_tasks to utilize ProgressBarManager for improved progress tracking.
- Modified tests to validate the new behavior functions
- Introduced a locking mechanism to ensure thread-safe job processing.
- Added `_jobs_processed` attribute to maintain consistent job indexing across multiple runs.
- Implemented `clear_jobs` method to reset job indices.
- Updated tests to verify exception handling and job indexing after clearing jobs.
- Adjusted Jupyter notebook tests for consistency in results retrieval.
…sted Transformation or Parallel procedures

- The apply_transforms() function has been updated to handle different types of transformations recursively:
  If transforms is a list, it recursively applies each transform in the list.
  If transforms is a Parallel instance, it recursively applies the transformations contained within the Parallel object.
  If transforms is a BaseGraphTransformation, it generates an execution plan (a list of coroutines), gets a description, and then runs the coroutines asynchronously using run_async_tasks().
  If transforms is none of the above, it raises a ValueError indicating an invalid type.
- Move apply_nest_asyncio to async_utils for better organization
- Updated the Parallel class to support transformations with improved type hints.
- Added unit tests
…e Executor

- nest_asyncio has internal checks for whether it is applied, rely on those
- Ensured apply_nest_asyncio is called in the Executor class to handle nested event loops.
- Removed optional progress bar parameter from process_futures.
- Updated calls to process_futures to handle progress bar updates directly.
- Cleaned up error handling logic for results in async processing.
- Replaced event loop checks with a unified async wrapper for scoring.
- Simplified error handling and callback management.
- Ensured compatibility with Jupyter environments by applying nest_asyncio directly.
@ahgraber ahgraber force-pushed the feature/async-executor branch from 4b7774f to ad79704 Compare September 8, 2025 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants